EVALUATION PLAN
VSAM of Massachusetts (VSAM)


I. EVALUATION FRAMEWORK

A. Introduction

This Evaluation Plan is designed to provide a framework for assessing the impact of VSAM program in the Salemwood School in Medford, Massachusetts, utilizing an outcome-based model.  The plan includes an overall Logic Model that identifies a long-term goal as well as interim or sub-goals against which progress can be measured.  The plan was developed by an Evaluation Committee, in consultation with evaluators Mindy Fried, Ph.D. and Mignon Duffy, M.A.  Evaluation Team members are:  Sandy Middleton, Director of Education Programs, VSAM, and Natalie Meibach, Education Program Coordinator, VSAM.


B. Who wants to know what and for what purpose?

In designing the evaluation, we first asked ourselves, "Who wants to know what and for what purpose?"  Initially, we approached this question broadly, considering the stakeholders for VSAM programs, in general, and why they would want evaluation data. We then focused more specifically on why key stakeholders would want this data for the Salemwood project.  The key stakeholders we identified include:
* the national VSA office
* VSAM 
* Private funders
* State/federal DOE
* Massachusetts Cultural Council
* School (Salemwood School in this case)
* Teachers
* Artists

Note:  See Appendix 1 for a detailed outline regarding each stakeholder's interest in evaluation, both for the Salemwood project and more generally, for other VSAM projects.


C. The Logic Model for Salemwood School Project:  Evaluation Goals and Indicators, and Program Components, or Inputs (see attached Logic Model)

The VSAM Logic Model - developed by the Evaluation Team - is a graphic representation of the relationship between the organization's programs (inputs), the interaction between the various program components, and the intended outcomes.  Generally speaking, a Logic Model looks something like the following diagram:  

Program 		>>>>      Sub-goals     <<< -- >>>>           Overarching Goals/Outcomes
(input/intervention)

To understand a Logic Model, one works backwards, first focusing on the long-term, or overall, goal, then on the sub-goals, and finally, on the program (i.e., input) which must be in place to achieve these goals.  This section details these components.

1. Long-term goals
The long-term, or overall goals for VSAM program at Salemwood School are the following:

* The Salemwood School integrates arts education into its curriculum and teaching practices, reflecting systemic changes and broader cultural acceptance of arts education in the schools; and
* The school and its arts education programs are a model for other similar programs/schools.


2. Sub-goals linked to VSAM overall goal 
VSAM's long-term goals are dependent on five sub-goals, which are more immediate aims of the program.  These sub-goals are:

GOAL # 1:  Children
All children, including those with disabilities, learn through the integration of arts education into the academic curriculum.

GOAL # 2:  Teachers
Teachers integrate arts education techniques into their own instructional practices.

GOAL # 3:  Administrative leaders
Administrative leaders support efforts to integrate arts education into the curriculum/infrastructure of the institution.

GOAL # 4:  Artists
Artists model best practices for teachers.

GOAL #5:  Collaboration between VSAM, Salemwood School and artists
The collaborative partnership between VSAM, the Salemwood School and the artists involved in the VSAM residencies is positive and productive.


3. Program Outcomes and Indicators
For each sub-goal, there are key program outcome areas, used to measure the extent to which goals are being achieved.  Within each of the program outcome areas we have identified indicators, or signs of program success, which are essentially the measures to be used in evaluating the impact of VSAM's work with Salemwood School.  The process of evaluation may also elicit unanticipated outcomes, which should also be incorporated into the evaluation process.    

4. Program components 
The Logic Model identifies "inputs," or program components that must be in place to support the achievement of goals.  Where relevant, the evaluation should seek to understand the relative influence of each program component on the program goals (e.g., placement of artists in the classroom versus providing professional development for teachers).


5. VSAM's Logic Model:  developing a theory of change (see Appendix 2)

How can the development of a Logic Model facilitate an understanding of how change occurs?
VSAM's Logic Model begins to address the theory of change implicit in the organization's work with schools, by identifying the relationships between and among the goals and their intended effect.   How does change occur for students at Salemwood School, as a result of exposure and engagement with arts education?  How do the above goals interface to result in the long-term goal?  
 
It is the complex interaction between and among more immediate goals that will fuel program success and further the long-term goal.  For example, there must be leadership from school administrators before artists are brought into the school to work with teachers and students.  With leadership from the administration, teachers are exposed to artists-in-residence in their classrooms.  This exposure allows teachers to learn from the artists, who model how to integrate arts education into the curriculum and pedagogy.  Students ideally reap the benefits, as teachers work with artists, and eventually continue this work with students after the artists have completed their residencies.  

Ultimately, the process of articulating a theory of change is an ongoing dialogue, which may have implications regarding what could and should be included in a program, as well as how a program is implemented, to achieve a desired goal.

VSAM's Logic Model 
In VSAM's Logic Model, the long-term goal is represented at the far right of the diagram because it can be achieved only with the achievement of the sub-goals.  The sub-goals - including those for the teachers, the students and the artists are represented as overlapping circles in the middle of the diagram within the administration goal.  These concentric circles reflect the interactivity between and among the goals, with change for students occurring within the school culture, comprised of administration support and teacher knowledge.  

The inputs or interventions on the left side of the Logic Model are program activities conducted by VSAM that must be in place to achieve any of the sub-goals or goals.  These include:  VSAM's collaborative relationship with artists and the school, VSAM hiring and training artists, VSAM's professional development for teachers, and ongoing contact with VSAM, teachers, artists and the school administration.


D. Sources of data and research methodologies

The key proposed sources of data include:
* data generated through program activities; and
* data generated through evaluation activities.

The proposed evaluation focuses on qualitative research methodologies (listed below, and in more detail in Evaluation Plan).  We have selected this approach because the numbers of individuals affected by the program intervention is relatively small, and we believe that we can maximize our understanding of the impact of the program through classic qualitative research methods such as focus groups, interviews and participant observation.



We will triangulate data from multiple sources, which may include the following:
* review of materials/curriculum generated by the artists;
* observations of artists working with children to assess the process of implementing the intervention;
* focus groups with teachers;
* interviews with administrators from the school; 
* review of VSAM's documentation of collaboration with partners;
* review of VSAM's documentation of training/professional development with teachers; and
* review of VSAM's teacher logs regarding student development resulting from the intervention.

Furthermore, we will maximize use of the available data regarding the program's effectiveness through the following means:

* Data generated through program activities 
We will support VSAM's program staff to collect data generated through program activities.  This may include documentation of partner collaboration, documentation of training/professional development for teachers, review of teacher logs regarding student developmental progress in relation to the arts curriculum, and teacher evaluations of VSAM professional development and working with resident artists.

* Evaluation-generated data 
We will collect data regarding the implementation of the program, as well as associated outcomes of the program on children, teachers, administrators and the collaboration itself between VSAM, the school and the artists.  Data collection methods may include interviews, focus groups and participant observation.

 
II. 
PROPOSED EVALUATION PLAN

This section identifies each sub-goal with program outcomes associated with the goal, and lists indicators, or measures, to be used in assessing progress towards reaching these goals.  It also includes the program components that impact each sub-goal.  Finally, research methods are outlined, with several data collection options presented.

SUB-GOALS, PROGRAM OUTCOMES, INDICATORS AND PROGRAM COMPONENTS THAT IMPACT EACH SUB-GOAL

GOAL # 1:  
All children (in the six kindergarten classrooms involved in the VSAM program), including those with disabilities, learn through the integration of arts education into the academic curriculum.

Program Outcomes and Indicators for Goal #1:  

1.a. General skill development resulting from participation in arts education
* Children will increase their listening skills;
* Children will increase their critical thinking skills;
* Children will increase their problem-solving skills;
* Children will increase their ability to work collaboratively in a group; and
* Children will increase their ability to articulate feelings and ideas.

1.b. Dance:  movement elements and dance skills
Students will:
* Learn key terms used in dance. 
* Identify and demonstrate basic locomotor movements (e.g., walk, run, hop, jump, leap, gallop, slide and skip), traveling forward, backward, sideward and diagonally, turning);
* Identify and demonstrate basic non-locomotor movements (e.g., bend, twist, stretch and swing);
* Creates shapes at low, middle and high levels;
* Develop strength, flexibility, balance, and neuromuscular coordination.
* Identify and begin to develop accuracy in moving to a musical beat and responding to changes in tempo.
* Demonstrate the ability to define and maintain personal space.
* Demonstrate movements in straight and curved pathways.
* Recognize and explore space, time, and force as three elements of movement and dance.
* Demonstrate partner skills of copying, leading, following and mirror imaging.
* Demonstrate ability to work in a group to learn and perform sequences of movement and simple dances.
* Develop and value a positive body image.

1.c. Dance:  Choreography
Students will:
* Explore and invent movement.
* Create a dance phrase with a beginning, middle and end.
* Create a dance phrase and then vary it, making changes in space, time and energy/force.
* Demonstrate the ability to work effectively along, with a partner, and in an ensemble.
* Demonstrate the following partner skills:  copying, leading and following, mirroring.
1.d. Dance as expression
Students will:
* Express how movements can show feelings, images, thoughts, colors, sounds, and textures.

1.e. Review of dance skills
* At the end of the artist residency, students are able to articulate and/or reflect through movement their understanding of all the above.


1.f. Music:  Singing
Students will:
* Learn key terms in music.
* Sing different types of music, alone and with others.
* Sing independently.
* Sing expressively.
* Sing from memory a variety of songs representing genres and styles from diverse cultures.
* Begin to learn how to sing partner songs, rounds and simple two-part songs, with and without accompaniment.
* Begin to learn how to sing in groups and respond to the cues of the resident artist.

1.g. Review of music skills
* At the end of the artist residency, students are able to articulate and/or reflect through music their understanding of all the above.

Program components that impact Goal #1:
* Artists in the classroom.
* Adaptive strategies used by artist.
* Teacher mentored by artists.
* Teacher continues to implement art education for students with VSAM support.


GOAL # 2:  
Teachers integrate arts education techniques into their own instructional practices.

Program Outcomes and Indicators for Goal #2:
2.a. Participation
* Teachers attend VSAM training.

2.b. Awareness
* Teachers understand and are able to articulate the usefulness of arts education in the classroom.

2.c. Skill development
* Teachers work collaboratively with artists during residency and are open to learning new skills and new ideas.
* Teachers develop their own skills in arts education (e.g., dance, music).
* Teachers engage in the practice of integrating the arts into other aspects of their curriculum outside of the residency.
* Teachers use material generated through the arts for public displays.

2.d. Leadership
* Teachers share what they learn from their experience with VSAM with other teachers at the school through leading in-services, etc.
* Teachers become public advocates for arts education.
* Teachers publicly display material generated from their activities in arts education.
* Teachers train others in arts education (share the model) in various venues.


Program components that impact Goal #2:
* Resident artists model arts education for teachers, as they work with children.
* Resident artists work directly with teachers to teach them how to integrate arts education into curriculum.
* Teachers participate in professional development on arts education.
* Ongoing involvement of arts specialist, music specialist, and librarian in VSAM program.
* Teachers share written evaluation materials regarding the effectiveness of the artists in achieving their goal with children, and the impact of arts education on children's learning.


GOAL # 3:  
Administrative leaders support efforts to integrate arts education into the curriculum/infrastructure of the institution.

Program Outcomes and Indicators for Goal #3

3.a. Participation in planning for integrating arts education into various aspects of school  
* Administration participates in short- and long-range planning to integrate arts education into the curriculum, pedagogy and school infrastructure.

3.b. Support for implementation of VSAM program 
* Administration agrees to implement VSAM program in six kindergarten classes.
* Administration asks VSAM to do professional development for all teachers in the school.
* Administration commits to purchasing Start with the Arts manual for all teachers who did not have a copy (VSAM gave it for free the six classes in program).
* Administration provides (at least partial) funding for arts education in school program.

3.c. Support for teachers to integrate arts education into curriculum
* Administration provides opportunities for teachers to share knowledge with school community, in various venues, including in-service days.
* Administration purchases 200+ children's books that support the Start with the Arts curriculum.

3.d. Administration takes leadership role in promoting arts education
* Administration collaborates with other organizations on arts education.
* Administration publicly advocates for arts education in the schools. 




Program Components that impact Goal #3:
* Administrators participate with VSAM in planning sessions the year leading up to program implementation
* Administrators participate in VSAM professional development
* VSAM supports the administration in engaging teachers to integrate arts education into curriculum

GOAL # 4:  
Artists model best practices for teachers.

Program Outcomes and Indicators for Goal #4

4.a. Artists are aware of the importance of arts education 
* Artists value the importance of integrating arts into the curriculum

4.b. Artists have the skills to bring arts education into the classroom
* Artists have skills to bring art into the classroom, work with children, work with children with disabilities, and work with teachers
* Artists are able to develop age-appropriate lessons that incorporate the Start with the Arts philosophy
* Artists adapt/incorporate elements of Start with the Arts curriculum into their lessons

4.c. Artists model best practices for teachers
* Artists model best practices for teachers

4.d. Artists mentor teachers 
* Artists mentor teachers to understand and implement model
*  Teachers adopt some of the practices modeled by artists

Program components that impact Goal #4
* Artists participate in VSAM orientation prior to working in the schools
* VSAM provides residency planning support to artists
* VSAM provides lesson planning support to artists
* VSAM provides ongoing support to teachers through various methods, including classroom observation followed by feedback sessions


GOAL #5:  
The collaborative partnership between VSAM, the Salemwood School and the artists involved in the VSAM residencies is positive and productive.

Program Outcomes and Indicators for Goal #5

5.a. Partners collaborate to implement pilot
* Partners meet regularly together to design and implement pilot.
* Partners (e.g., school administration and teachers, artists and VSAM) clarify roles and responsibilities.
* Teachers, artists and VSAM maintain regular contact via meetings, phone and email contact to share progress of program, obstacles faced and strategies to overcome obstacles.

5.b. Partners evaluate program
* Partners support the work of VSAM and independent evaluators to design and implement an evaluation of the program.

5.c. Partners work together to implement the project beyond the pilot
* Partners work together to further spread pilot to other parts of school, getting buy-in from key stakeholders.
* Partners work together to find additional funding for the program.

5.d. Partners work together to bring the model to other schools outside of the Salemwood School
* Partners work together to get and use evaluation data regarding the impact of the program at Salemwood School.
* Partners work together to share the model beyond Salemwood School.


Program components that impact Goal #5:
* VSAM takes the lead in pulling partners together to plan and implement program
* VSAM maintains ongoing contact with the school administration regarding implementation issues, including teacher buy-in and support, artist effectiveness and response to interventions from children 
* VSAM maintains ongoing contact with artists to assess the effectiveness of intervention


RESEARCH METHODS
As stated earlier, the evaluation of VSAM's program at Salemwood School focuses on qualitative research methods.  These include:  documentation of meetings and training sessions, review and analysis of materials generated by the program, focus groups with teachers, interviews with artists, observations of artists, teachers and children, and review of teacher logs regarding student development that results from program intervention.  Some of the data will be collected by VSAM staff, with support from independent evaluators who will generate protocols to guide data collection.  Other data will be collected by independent evaluators, to provide a more neutral assessment of the effectiveness of the program, and its impact on intended outcomes.
A. VSAM-generated data
The following data collection methods will be conducted by VSAM.  Where appropriate, the independent evaluators will work with VSAM to develop protocols to collect data.

A.1. Partners
* documentation of meetings with partners (e.g., VSAM, school administrators and teachers and artists)
VSAM will document meetings of partners to better understand the effectiveness of the partnership among key stakeholders.  The kind of information that should be captured in this documentation includes:  clarification of roles and responsibilities among the partners, level of support from school officials, level of buy-in from teachers, factors that may promote the VSAM program, obstacles that may arise to effectively implementing the VSAM program, indicators of developing support to expand the program beyond the initial pilot sites.  


A.2. School
* written materials 
VSAM will analyze written materials that are generated by Salemwood School regarding the program, to assess the level of support and the extent to which it is being promoted within the school culture.


A.3. Teachers
* documentation of meetings and training sessions with teachers
VSAM will document sessions with teachers, including formal and informal meetings, as well as training sessions.  The focus will be on assessing teacher interest and buy-in, their working relationship with artists, and the development of their skills to utilize the arts curriculum after the artist completes her/his residency.

* review of teacher logs
VSAM will review teacher logs (e.g., notes) regarding student development resulting from the intervention.


A.4. Artists
* review of artists' curriculum 
VSAM will review artists' curriculum to understand how various activities are designed to promote the goals and objectives of the program, and to gauge the responsiveness of artists to individual needs of children.

* documentation of ongoing contact with artists
VSAM will document ongoing contact with artists, focusing on their assessment of the impact (e.g., successes, benefits) of the program, the obstacles faced and the strategies they use to overcome obstacles.

* review of artists' logs
VSAM will review artists' logs (e.g., notes) regarding student development resulting from the intervention.

* classroom observations 
VSAM will observe classrooms, focusing both on the effectiveness of the artists, as well as the impact of the program on the children.

B. Evaluation-generated data

B.1. Material review
Independent evaluators will review materials generated through the VSAM program from all potential sources, including school administrators, teachers, artists, and children.  We will conduct content analysis of these materials to identify themes relevant to achieving the overall goal for the program:  systemic change within the school to facilitate the integration of arts education in the curriculum and teaching practices.



B.2. Teachers
* focus groups 
Independent evaluators will conduct focus groups with the six kindergarten teachers at two intervals throughout the intervention period.  The focus of the interviews will be on assessing the extent to which teachers support the use of arts education in the classroom.  We will investigate any changes in attitude and behavior regarding support for and use of arts education throughout the intervention period, and plans to incorporate arts education in their learning practices after the artists' residencies are completed.  This assessment will include a focus on:  teacher interest and attitude towards the arts curriculum, the perception of teachers regarding the impact of the arts curriculum on the children in their classrooms, the level of skill that teachers develop during the intervention period, the intent of teachers to carry on the arts curriculum beyond the intervention period, the extent to which teachers actually do carry on the arts curriculum beyond the intervention period.


B.3. Artists
*  Interviews 
Independent evaluators will interview artists to better understand their goals and objectives with children and teachers, including their skill level working with kindergarteners and kindergarten teachers, their abilities to work in the school setting, the obstacles they perceive and/or experience in working in this setting, and their creativity in addressing potential obstacles.

* review artists' materials, including curriculum and teaching aides
Evaluators will review artists' lesson plans to understand the goals and objectives of artists in their work with children as well as teachers, focusing on the extent to which these goals/objectives match (or conform) to the desired goals by the national VSA Arts (and other critical sources), as presented in orientation and ongoing communications by the VSAM staff.

* observation of artists in classrooms
Independent evaluators will observe artists in the classroom to assess their effectiveness with the children and the teachers, focusing on the extent to which these goals/objectives match (or conform) to the desired goals by the national VSA Arts (and other critical sources) in practice.


B.4. Administration officials
* interviews with school administrators
Independent evaluators will interview school administrators at two intervals throughout the period of intervention to assess their level of support, the extent to which they are developing as advocates and leaders in promoting arts education, their efforts to encourage teachers to continue utilizing arts education, the obstacles they face and how they address them.



Mindy Fried, Ph.D and Mignon Duffy, M.A.,, Evaluator
January 7, 2003


 

Appendix 1

Stakeholder
Why want evaluation data re: Salemwood program?

Why want evaluation data (more generally for VSAM projects)?

1. National office
* to identify best practices to share with other affiliates
Please note that the national office does not provide direct funding for the Salemwood project, although it does provide indirect funding (e.g., via VSAM operating expenses).
* to identify best practices to share with other affiliates;
* to provide data regarding extent to which programs are accomplishing organizational mission to meet the needs of children, especially those with disabilities; and
* to justify funding for affiliate programs

2. VSAM

VSAM wants evaluation data for the following factors, as well as for those listed in the next column (i.e., data desired more generally for VSAM projects).

* to know the extent to which programming is achieving goal (making a difference for students and teachers in six kindergarten classes);
* to understand process of program implementation and its effect on outcomes (e.g., What are the constants in the residency program?  What do artists do to establish rapport with a teacher, students in the class, etc.);
* to get information that will inform future programming (e.g., to inform structure of residencies, training, etc.);
* to get support and buy-in from Salemwood School, including funding for the program next year; and
* to have data regarding the impact of the program on kindergarteners, which may allow VSAM to market services to networks of child care providers.
* to know the extent to which programming is achieving goal (making a difference);
* to assess extent to which program is cost-effective;
* to educate parents regarding the value of program;
* to provide better support and professional development to artists and teachers in the program;
* to expand impact of the program on the whole school; and
* to effect broader policy change (e.g., school reform).

3. Private funders
There is no funding for this project from private sources, but VSAM would like to use the project as a model for future funding.

* to justify investment in effective programming;
* to justify investment of resources to fund inclusion, so that students of all abilities have access to programming; and
* to provide publicity for funding organizations.

5. State/Federal DOE
* to justify funding;
* to provide data regarding the achievement of student and teacher goals, especially regarding students with disabilities; and
* good data on program effectiveness helps government bodies make thoughtful decisions in awarding contracts in this competitive environment.
* to justify funding;
* good data on program effectiveness helps government bodies make thoughtful decisions in awarding contracts in this competitive environment; and
* to provide data regarding the achievement of student and teacher goals, especially regarding students with disabilities.

6. Massachusetts Cultural Council

* to justify funding;
* to encourage school collaboration with nonprofits who are interested in understanding how programs are working; and
* to meet the goals of constituents, especially regarding opportunities for people with disabilities.


Stakeholder
Why want evaluation data re: Salemwood program?

Why want evaluation data (more generally for VSAM projects)?

4. School
VSAM wants evaluation data for the following factors, as well as for the factors listed in the next column (with some overlap). 
* to support efforts to achieve the mission (and name) of this Arts magnet school;
* to get buy-in from teachers and administrators at Salemwood School (to support the commitment of the Superintendent and Asst. Superintendent); and
* to provide evidence to parents that the Salemwood School is providing quality Arts education in the curriculum/

Note:  School is not investing funds for this current fiscal year, but has potential to get matching grant from MCC next year to continue program.  


* to justify support for program by providing data regarding its effectiveness (schools as "customers" who need information about the quality of services provided);
* to validate VSAM impact as part of whole school reform plan;
* to help parents understand how the arts in education is a means to help them become better allies/advocates for their children.

7. Teachers

* to get teacher buy-in by providing data regarding effectiveness of program;
* to elicit support and interest in professional development from VSAM and MassArt; and
* to attract new teachers who are committed to fully integrated arts education program.
* to identify best practices; and
* to validate the importance of the work they are engaging in as they integrate arts into broader curriculum.

8. Artists

* to identify best practices;
* to validate the importance of their work; and
* to provide feedback regarding the extent to which they are achieving their goals with children and teachers.
* to identify best practices;
* to validate the importance of their work; and
* to provide feedback regarding the extent to which they are achieving their goals with children and teachers.





Appendix 2:  VSAM Logic Model
(see separate file)








 
 





16


